Methods and Apparatus for Coordinated Lens and Sensor Motion

ABSTRACT

In exemplary implements of this invention, a lens and sensor of a camera are intentionally destabilized (i.e., shifted relative to the scene being imaged) in order to create defocus effects. That is, actuators in a camera move a lens and a sensor, relative to the scene being imaged, while the camera takes a photograph. This motion simulates a larger aperture size (shallower depth of field). Thus, by translating a lens and a sensor while taking a photo, a camera with a small aperture (such as a cell phone or small point and shoot camera) may simulate the shallow DOF that can be achieved with a professional SLR camera. This invention may be implemented in such a way that programmable defocus effects may be achieved. Also, approximately depth-invariant defocus blur size may be achieved over a range of depths, in some embodiments of this invention.

RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application Ser.No. 6116880, filed Apr. 13, 2009, the entire disclosure of which isherein incorporated by reference.

FIELD OF THE TECHNOLOGY

The present invention relates to cameras.

BACKGROUND

When taking a photograph, it is sometimes desirable to have a shallowdepth of field (DOF). This allows one to achieve artistic effects, inwhich a portion of the scene is in focus and the remainder of the sceneis out of focus. For example, one may use a shallow DOF so that a flowerin a scene appears in sharp focus and the more distant backgroundappears out of focus. This prevents the flower from being lost againstthe background.

Depth of field depends on a number of factors, including the aperturesize of the camera. The larger the aperture, the shallower is the DOF.

Shallow DOF (large aperture) may be achieved with professional SLRcameras. Unfortunately, less expensive cameras (such as cell phonecameras and small point and shoot cameras) typically have small lensapertures that cannot achieve such shallow DOF, and thus cannot createthe same artistic defocus effects that may be achieved with aprofessional SLR camera.

SUMMARY

In exemplary implements of this invention, a lens and sensor of a cameraare intentionally destabilized (i.e., shifted relative to the scenebeing imaged) in order to create defocus effects. That is, actuators ina camera move a lens and a sensor, relative to the scene being imaged,while the camera takes a photograph. This motion simulates a largeraperture size (shallower depth of field). Thus, by translating a lensand a sensor while taking a photo, a camera with a small aperture (suchas a cell phone or small point and shoot camera) may simulate theshallow DOF that can be achieved with a professional SLR camera.

This invention may be implemented in such a way that programmabledefocus effects may be achieved.

Also, approximately depth-invariant defocus blur size may be achievedover a range of depths, in some embodiments of this invention.

It is helpful to compare this invention to the conventional technique ofimage stabilization. Image stabilization involves moving either a lensor a sensor (but not both) of a camera in order to compensate for motionof the camera. It stabilizes the image, i.e., prevents the image frombeing defocused as a result of camera movement. In contrast, inexemplary implementations of this invention, both a lens and a sensor ofa camera (rather than just one of them) are moved at the same time. Thepurpose of this coordinated motion is to destabilize the image, i.e., tointentionally create defocus effects and to simulate a shallower depthof field. The velocities and direction of motion of the lens and sensormay be selected in such a way as to control the defocus effects that areachieved.

This invention may be implemented as a camera that includes one or moreactuators for causing a lens and a sensor of said camera, but not thecamera as a whole, to move relative to the scene being imaged, at thesame time that the camera captures an image. Furthermore: (1) the planeof said sensor, the plane of said lens, the direction of motion of saidsensor and the direction of motion of said lens may all be substantiallyparallel to each other; or (2) the plane of said sensor may besubstantially parallel to the plane of said lens but not substantiallyparallel to the direction of motion of said lens. Also, (3) said one ormore actuators may be adapted for moving said lens and said sensor insuch a way as to simulate a larger aperture size than the actualaperture size of said lens, (4) said one or more actuators may beadapted for moving said lens and said sensor in such a way as to achievea substantially depth-independent defocus blur size over a range ofdepths, (5) said substantially depth-independent defocus blur size maybe achieved over a range of depths while said lens and said sensortravel at substantially constant velocities, which range extends betweenthe depth at which said lens and said sensor would capture an in-focusimage while stationary and the depth at which, if a pinhole weresubstituted for said lens, said pinhole and said sensor would capture anin-focus image while traveling at said velocities. Furthermore: (6) atleast one of said actuators may be a stepper motor, (7) at least one ofsaid actuators may be piezoelectric, (8) at least one of said actuatorsmay be ultrasonic, (9) at least one of said actuators may be furtheradapted for moving at least one lens or sensor of said camera undercertain circumstances, in such a way as to compensate for motion of saidcamera, (10) said one or more actuators may be adapted for moving saidlens and said sensor, each at a constant velocity for a substantialportion of the total time of said movement, (11) said one or moreactuators may be adapted for moving said lens and said sensor, each at avelocity that varies substantially during a substantial portion of saidmovement, which portion does not include the initial acceleration orfinal deceleration that occur during said movement, (12) motions of saidlens or said sensor may be circular, elliptical, hypocycloidal orspiral, (13) the image may be captured during a single exposure, and(14) movement of said lens and said sensor may be programmable.

This invention may be implemented as a method in which at least oneactuator of a camera moves a lens and a sensor of a camera, but not thehousing of a camera, relative to the scene being imaged, at the sametime that the camera captures an image. Furthermore, (1) said movementof said lens and said sensor may be programmable; (2) said lens and saidsensor may be moved in such a way as to simulate a larger aperture sizethan the actual aperture size of said lens, (3) at least one saidactuator may move said lens and said sensor in such a way as to achievea substantially depth-independent defocus blur size over a range ofdepths, which range extends between the depth at which said lens andsaid sensor would capture an in-focus image while stationary and thedepth at which, if a pinhole were substituted for said lens, saidpinhole and said sensor would capture an in-focus image while travelingat said velocities, and (4) at least one actuator of said camera maymove at least one lens or sensor of said camera in such a way as tocompensate for motion of said camera.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one color photograph.Copies of this patent or patent application publication with colorphotograph(s) will be provided by the Office upon request and payment ofthe necessary fee.

In the Detailed Description of the invention that follows, referencewill be made to the following drawings:

FIG. 1 is an isometric view of a prototype of this invention.

FIG. 2 is a side view of that prototype.

FIG. 3 is a view of a computer and USB cable used in a prototype of thisinvention.

FIG. 4 is a diagram showing lens-based focusing.

FIG. 5 is a diagram that illustrates how a smaller aperture results in asmaller defocus blur.

FIG. 6 is a diagram that shows how a pinhole may be used to create anall-in-focus image.

FIG. 7 is a diagram that shows a pinhole being moved during an exposure.

FIG. 8 is a diagram that shows both a pinhole and a sensor being movedduring an exposure, in an illustrative implementation of this invention.

FIG. 9 is a diagram that shows both a pinhole and a sensor being movedduring an exposure, in an illustrative implementation of this invention.

FIGS. 10A and 10B are charts that depict the size of defocus blur over arange of distances, in an illustrative implementation of this invention.

FIG. 11 is a diagram that shows a pinhole and a sensor being movedduring an exposure, where the movement of the pinhole is not parallel tothe alignment of the sensor, in an illustrative implementation of thisinvention.

FIG. 12 is an all-in-focus photograph taken with a static lens with anf/22 aperture.

FIG. 13 is a photograph that is focused on the closest (front) toy inthe scene. The photo was taken with a lens with an f/22 aperture, whichlens was translated 10 mm during the exposure, in an illustrativeimplementation of this invention.

FIG. 14 is a photograph that is focused on the middle toy in the scene.The photo was taken with a lens with an f/22 aperture, which lens wastranslated 5 mm during the exposure, in an illustrative implementationof this invention.

FIG. 15 is a photograph that is focused on the furthest (back) toy inthe scene. The photo was taken with a lens with an f/22 aperture, whichlens was translated 10 mm during the exposure, in an illustrativeimplementation of this invention.

FIG. 16 is an all-in-focus photograph of mirror balls taken with astatic lens with an f/22 aperture.

FIG. 17 is a photograph of mirror balls taken with a static lens with anf/2.8 aperture.

FIG. 18 is a photograph of mirror balls with the virtual focal plane isin the center, taken with a lens with an f/22 aperture. A lens andsensor were translated during the exposure, in an illustrativeimplementation of this invention.

FIG. 19 is a photograph of mirror balls taken with a lens with an f/2.8aperture. A lens and sensor were translated during the exposure, in anillustrative implementation of this invention.

FIG. 20 is a photograph of mirror balls taken with a lens with avertical slit aperture. A lens and sensor were translated during theexposure, in an illustrative implementation of this invention.

FIG. 21 is a photograph of toy figures taken with a static lens with anf/2.8 aperture.

FIG. 22 is a photograph of toys. A lens and sensor were translatedduring the exposure, in an illustrative implementation of thisinvention.

FIG. 23 is a photograph of toys taken with a lens with a horizontal slitaperture. A lens and sensor were translated during the exposure, in anillustrative implementation of this invention.

FIG. 24 is a photograph of toys. For this photo, Richardson-Lucydeconvolution results in an approximately all-in-focus image, in anillustrative implementation of this invention.

DETAILED DESCRIPTION

In exemplary implementations of this invention, actuators in a cameramove a lens and a sensor of a camera, relative to the scene beingimaged, while the camera takes a photograph. This motion simulates alarger aperture size (shallower depth of field).

FIGS. 1 and 2 are perspective views of a prototype of this invention.FIG. 3 is a side view of this prototype. In this prototype, a sensor 1and a lens 2 are mounted on a pair of linear translation stages 4 and 5.Stepper motors make the linear translation stages (and thus the sensorand lens) move.

In this prototype, the sensor 1 is the sensor on a 12.2 megapixel Canon®EOS Digital Rebel XSi camera. The lens 2 is a Nikkor® 50 mm f/1.8D lenswith manual aperture control. In addition, a second diverging lens 3 isplaced behind the Nikkor lens, in order to form a focused image on thesensor. The camera and lens are enclosed in a box (not shown in FIGS.1-2) to prevent stray light from reaching the sensor. External steppermotors 6 and 7 cause rods 9, 10 to move, driving the translation stages4 and 5. The rods 9, 10 are parallel to each other. A circuit board 8 isemployed to control the stepper motors. Exposures are timed to occuroutside of the stepper motor ramp-up and ramp-down phases. In thisprototype, the translation stages allow a total displacement of 4 cm andtypical exposures may range from 5 to 30 seconds. In this prototype, acomputer 31 (shown in FIG. 3) is used to control the camera. Thecomputer is connected to the camera with a USB cable 32.

Before discussing how the present invention works, it is helpful tobriefly review (a) focusing with a conventional static lens, (b)all-in-focus imaging with a static pinhole, and (c) defocus caused bymoving the pinhole, but not the sensor, relative to the scene beingimaged.

FIG. 4 illustrates focusing with a conventional, static lens. In FIG. 4,scene objects at a certain depth from the lens (i.e., in scene plane 41)appear in sharp focus on the sensor. Point A is at that depth; thus itis imaged as a focused point of light A′ on the sensor plane. Incontrast, point B is not at that depth; thus it is imaged as a defocusblur B′ on the sensor plane. The size of the defocus blur isproportional to the size of the aperture. For example, as shown in FIG.5, a smaller aperture causes a smaller defocus blur.

FIG. 6 illustrates how a static pinhole camera may be employed forall-in-focus imaging. FIG. 6 is a simple ray diagram for a pinholecamera, where scene points A and B are projected to points A₀′ and B₀′on the sensor, respectively. Since the pinhole selects a single ray fromeach scene point, the entire scene appears focused on thesensor—irrespective of the distance of a given point from the pinhole.

FIG. 7 illustrates the effect of moving the pinhole, but not the sensor,relative to the scene being imaged. As shown in FIG. 7, the pinhole ismoving with velocity v_(p) relative to the scene being imaged and thesensor. Light from points A and B are projected as defocus blurs A′ andB′, respectively, on the sensor plane.

In an illustrative implementation of this invention, both a sensor and apinhole are moved at the same time relative to the scene being imaged.FIGS. 8 and 9 illustrate such a configuration. The pinhole is translatedat velocity v_(p) and the sensor is translated with velocity v_(s), eachrelative to the scene being imaged. The directions of motion areparallel. In this configuration, the velocities of the sensor and lensmay be selected in such a way that (A) the acquired image is focused ona specific scene plane at a distance d_(a) from the pinhole, and (B)points in other scene planes are defocused. As the pinhole moves from P₀to P₁, the image of point A shifts from A₀′ to A₁′. To focus on theplane containing point A, the sensor must be translated such that A₀′and A₁′ overlap. This occurs when the sensor displacement t_(s) is givenby:

${t_{s}\left( {1 + \frac{d_{s}}{d_{p}}} \right)}t_{p}$

where t_(p) is the pinhole displacement, d_(s) is the distance betweenthe pinhole and d_(a) is the distance between the pinhole and the sceneplane containing point A.

Since this applies to any point on the plane at a distance d_(a) fromthe pinhole, a parallel translation of the pinhole and sensor may beemployed to produce an image focused at d_(a). Specifically, in thispinhole configuration, if the pinhole moves a constant velocity v_(p)during the exposure, then the sensor must translate with a constantvelocity

$\begin{matrix}{v_{s} = {\left( {1 + \frac{d_{s}}{d_{a}}} \right)v_{p}}} & (1)\end{matrix}$

in order for the acquired image to be focused on the scene plane atdistance d_(a) from the pinhole.

Note that, if velocities are selected in this manner in thisconfiguration, points at a distance other than d_(a) from the pinholewill appear defocused.

For example, consider the scene point B at a distance d_(b) from thepinhole plane, in the configuration shown in FIGS. 8 and 9. The image ofthis point moves from B₀′ to B₁′ as the pinhole moves from P₀ to P₁. Thetotal displacement t_(b) of the image of B as the pinhole translatesover a distance t_(p) is given by

$t_{b} = {d_{s}{{\frac{1}{d_{b}} - \frac{1}{d_{a}}}}{t_{p}.}}$

Thus, the parallel motions of the sensor and pinhole, relative to thescene being imaged, reduce the depth of field of the optical setup. Forsuch a pinhole configuration, the diameter of the circle of confusion isthen given by

$\begin{matrix}{c_{P} = {\frac{d_{s}}{d_{a}}\left( \frac{d_{\Delta}}{d_{a} + d_{\Delta}} \right)t_{p}}} & (2)\end{matrix}$

where d_(Δ)=d_(b)−d_(a) is the distance from the plane of focus.

The term “pinhole shift” refers to a sensor and pinhole being moved,relative to the scene being imaged, while an image is captured.

Now consider how the parallel motions of a sensor and pinhole may beused to simulate the optics of an ideal thin lens.

A thin lens is governed by the thin lens equation

$\begin{matrix}{{\frac{1}{f_{T}} = {\frac{1}{u} + \frac{1}{v}}},} & (3)\end{matrix}$

where f_(T) is the focal length of the thin lens, u is the objectdistance, and v is the image distance.

Rearranging this expression and comparing with Equation 1 (whered_(a)=u, and d_(s)=v), the virtual focal length f_(P) for pinhole shiftis given by

$\begin{matrix}{f_{P} = {\left( \frac{v_{p}}{v_{s}} \right)d_{s}}} & (4)\end{matrix}$

The diameter C_(T) of the circle of confusion for a thin lens is givenby the relation

$\begin{matrix}{{c_{T} = {\frac{f_{T}}{d_{a} - f_{T}}\left( \frac{d_{\Delta}}{d_{a} + d_{\Delta}} \right)A}},} & (5)\end{matrix}$

where A is the aperture diameter of the thin lens.

Combining Equation 5 with the thin lens equation:

$c_{T} = {\frac{d_{s}}{d_{a}}\left( \frac{d_{\Delta}}{d_{a} + d_{\Delta}} \right){A.}}$

Comparing this result with Equation 2, it is clear that the totaldisplacement t_(p) for pinhole shift must be equal to the aperture sizeA in order to replicate the circle of confusion for a given thin lens.Thus, the virtual f-number (the ratio of the virtual focal length to thevirtual aperture size) for pinhole shift is given by

$\begin{matrix}{N_{P} = {\frac{f_{P}}{t_{p}} = {\left( \frac{v_{p}}{v_{s}} \right){\left( \frac{d_{s}}{t_{p}} \right).}}}} & (6)\end{matrix}$

Thus, according to principles of this invention, the synchronizedtranslation of a pinhole and sensor allows a pinhole camera to replicatethe effect of an arbitrary thin lens. Adjusting the relative translationvelocities {v_(p), v_(s)} and total displacements {t_(p), t_(s)} of thepinhole and sensor allows the synthesis of a thin lens with focal lengthf_(T) and f-number N_(T).

This result can also be understood by interpreting a thin lens as auniform array of translated pinholes and prisms. Under this model, theimage detected by the sensor is a linear superposition of the individualimages formed by each shifted pinhole-prism pair. A local segment of thethin lens with focal length f_(T), located a distance t_(p) from theoptical axis, acts as a pinhole followed by a prism that produces aconstant angular deflection α=t_(p)/f_(T). Under the paraxialapproximation, the prism effectively translates the resulting pinholeimage by a distance t_(s) given by

$t_{s} = {{{- \alpha}\; d_{s}} = {\frac{d_{s}t_{p}}{f_{T}} = {\left( {1 + \frac{d_{s}}{d_{a}}} \right){t_{p}.}}}}$

This translation t_(s) is identical to the sensor translation given byEquation 1.

Thus, in illustrative implementations of this invention, thesynchronized translation of a pinhole and the sensor effectively createsa “thin lens in time”, where the pinhole translation scans the apertureplane and the sensor translation replaces the action of the localprisms.

It is often desirable to use a lens rather than a pinhole, in order toavoid loss of light and diffraction associated with pinholes. Thus, insome implementations of this invention, a lens with a finite aperturesize is used instead of a pinhole. That is, a sensor and a lens with afinite aperture are moved, relative to the scene being imaged, at thesame time that an image is captured.

The above analysis (regarding coordinated translation of a pinhole andsensor) can be extended to coordinated translation of a lens (with afinite aperture size) and a sensor. A pinhole can be interpreted as athin lens with an infinitely-small aperture located at the opticalcenter. The virtual focal length and f-number for such a configurationis given by Equations 4 and 6 and the point spread function (PSF) is abox function for 1D motions (or a pillbox for 2D) corresponding to thecircle of confusion in Equation 2. As the aperture size increases, theoverall PSF h^(L) is a combination of the virtual PSF due to pinhole andsensor translation and the physical PSF due to the lens aperture. Theoverall PSF is given by

h _(f) _(T) _(,N) _(T) _(,f) _(P) _(,N) _(P) ^(L)(d)=h _(f) _(T) _(,N)_(T) ^(T)(d)*h _(f) _(P) _(,N) _(P) ^(P)(d),  (7)

where h^(T) is the physical PSF of the thin lens, h^(P) is the virtualPSF due to sensor and lens translation, and d is the distance of thepoint source from the lens plane.

Thus, in exemplary implementations of this invention, translating afinite aperture lens synchronized with the sensor results in thecreation of a second virtual lens, and the effective PSF of theresulting system is the convolution of the PSFs of the real and virtuallenses.

In exemplary implementations of this invention, a special case occurswhere the real and virtual focal lengths are matched (i.e.,f_(T)=f_(P)). In that special case, a shifting lens and sensor behavesvery similar to a static lens of the same focal length, but with alarger effective aperture size (or smaller effective f-number). For thissituation, a single plane is in focus and the size of the circle ofconfusion rapidly increases for scene points located away from thisplane. The increased effective aperture size yields a depth of fieldthat is shallower than what is obtained by either a static lens withf-number N_(T) or a translating pinhole configuration with f-numberN_(P). The overall f-number N_(L) of a shifting lens and sensor is givenby

${\frac{1}{N_{L}} = {\frac{1}{N_{T}} + \frac{1}{N_{P}}}},$

where N_(P) is the virtual f-number given by Equation 6. Even though theeffective aperture size is increased, the total light entering thecamera during the exposure remains identical to that allowed by theunmodified physical aperture.

The effective PSF of a shifting lens and sensor is the convolution ofthe real and virtual PSFs. Thus, according to principles of thisinvention, limitations of the physical PSF due to the lens can beaddressed by engineering an appropriate virtual PSF by selectingappropriate motions for a lens and sensor. The component h^(P)(d)depends on the relative velocities and paths (in 2D) of the lens andsensor as they translate. These parameters may in some cases easier tocontrol than the optical elements within the lens. In exemplaryimplementations of this invention, coordinated translation introducesadditional blur. As a result, according to principles of this invention,synchronized translation of a lens and sensor can be applied toattenuate high-frequency components in the physical PSF and improve theoverall bokeh.

In the special case where the real and virtual focal lengths are matched(i.e., f_(T)=f_(P)), the size of the defocus blur due to shifting a lensand sensor is approximately equal to the sum of the size of the defocusblur due to (1) the fixed lens, and (2) pinhole shift. FIG. 10A is achart that illustrates this. It plots the size of the circle ofconfusion for (a) a fixed lens, (b) pinhole shift, and (c) the combinedcase effect of shifting a lens and sensor. In FIG. 10A, the defocusenhancement is achieved using a lens with focal length 50 mm, aperture15 mm, focused at 8 m, and total lens displacement of 10 mm. As shown inFIG. 10A, in the special case where the real and virtual focal lengthsare matched (i.e., f_(T)=f_(P)), the overall size of the combined circleof confusion is approximately equal to the sum of the two cases (fixedlens and pinhole shift), and the depth of field is shallower for thecombination.

In an exemplary implementation of this invention, the real and virtualfocal lengths (i.e., f_(T)=f_(P)) may be matched in order to enhance thedefocus (bokeh) effect.

The more general case where f_(T)≠f_(P) results in a setup that cannotbe duplicated with only a single fixed lens and sensor. In this case thetwo focusing mechanisms do not focus at identical planes. As a result,no single plane is focused on the sensor and the effective PSF for anyscene depth is the convolution of the two individual PSFs for thatdepth. If d_(a) ^(T) and d_(a) ^(P) are the two in-focus planes for aphysical lens and pinhole shift, respectively, then the size of thecombined circle of confusion is approximately constant for all planesthat lie between them, as shown in FIG. 10B. This results in adepth-invariant blur size for the specified range of scene distances. Anapproximately all-in-focus image may be obtained by deconvolving theconstant blur kernel.

Thus, in exemplary implementations of this invention, in the case wheref_(T)≠f_(P), synchronous translation of a lens and a sensor may beemployed to capture an approximately depth-invariant blur size over arange of distances between two planes d_(a) ^(T) and d_(a) ^(P), whered_(a) ^(T) and d_(a) ^(P) are the two in-focus planes for a physicallens and pinhole shift. FIG. 10B illustrates the results of an exampleof such a configuration, with the lens focused at 20 m, and a 15 mmtotal lens displacement. In the example shown in FIG. 10 B, thecumulative blur size is approximately a constant for all distances inthe range of 8 m to 20 m.

In the situation where f_(T)≠f_(P), the PSF generally varies with depth(even though the size of the circle of confusion is invariant to thedepth). However, there is an exception to this general rule: If the PSFsfor both the real and virtual focusing mechanisms have a Gaussian shape,then translation of the sensor and lens may be used to obtain an overallapproximately depth-invariant Gaussian PSF for the combined setup. Thus,this invention may be implemented in such a way as to capture anapproximately depth-invariant Gaussian PSF, in the special case of aGaussian PSF for both the real and virtual focusing mechanisms.

The above discussion considered only situations where the sensor isparallel to the lens or to the motion direction of the pinhole. Adifferent situation, where the sensor and lens are static and notparallel to one another, is well understood by the Scheimpflugprinciple. The plane of focus for such a setup is not parallel to eitherthe lens or the sensor, and passes through the line of intersectionformed by the extended planes containing the lens and the sensor asshown in FIG. 11A.

The Scheimpflug principle cannot be reproduced exactly by shifting thepinhole/lens and the sensor as discussed above. This is because thevirtual focal length for the pinhole shift configuration, as shown inEquation 4, is a function of the pinhole-sensor separation d_(s). Whilethis does not affect the case where the sensor is parallel to thedirection of the pinhole motion, the virtual focal length varies overthe surface of a tilted sensor, thus violating the traditionalScheimpflug principle.

However, in an illustrative implementation of this invention, similarresults are obtained using a translating pinhole as shown in FIG. 11B.The sensor is tilted at an angle α. The sensor and lens move in paralleldirections. Two points C and D focus on the image sensor over time. Thegeometric relationship between these points is given by:

$\frac{d_{c}}{d_{c} + d_{c^{\prime}}} = {\frac{d_{d}}{d_{d} + d_{d^{\prime}}} = {\frac{t_{p}}{t_{s}} = {\frac{v_{p}}{v_{s}}.}}}$

This gives the relation

${\frac{d_{c}}{d_{c^{\prime}}} = \frac{d_{d}}{d_{d^{\prime}}}},$

which implies that the line joining in-focus points C and D is parallelto the sensor (due to similar triangles). The setup focuses on a planethat is parallel to the sensor. The exact plane of focus depends on theratio of the sensor velocity to the pinhole velocity v_(s)/v_(p), andEquation 1 can be used to find it.

In an exemplary implementation of this invention, a translating lens canbe used in place of the translating pinhole in FIG. 11B. A lens parallelto the sensor also focuses on a plane parallel to the sensor (the exactplane depends on the focal length of the lens). Once again, either (a)the virtual focal length can be matched to the physical focal length toenhance the defocus (bokeh), or (b) they may be kept different toproduce a depth-invariant blur size across a scene.

The photographic results that may be achieved by this invention arestriking

FIGS. 12 through 15 are photographs taken by a prototype of thisinvention. They are basically pinhole images, taken using a lens stoppeddown to an f/22 aperture (which approximates a pinhole). In theseFigures, toy figures are arranged at different depths from the camera,with depth increasing from right to left. FIG. 12 is an all-in-focusimage, whereas in FIGS. 13, 14 and 15 only a portion of the sceneappears in focus. FIG. 12 was taken while the lens and sensor werestatic. The photographs in FIGS. 13, 14 and 15 were captured while thesensor and lens were moving relative to the scene. For the photos inFIGS. 13, 14 and 15, the lens was translated 10 mm, 5 mm and 10 mm,respectively, during exposure. In these Figures, the front, middle andback figures, respectively, appear in sharp focus. FIGS. 13, 14 and 15are examples of how shallow depth of field may be achieved by moving asensor and lens during exposure.

In illustrative implementations of this invention (in which a sensor anda lens move relative to the scene), virtual focal length may be variedby adjusting the velocity ratio as per Equation 4, allowing variousscene planes to be brought into focus in the different photos. Thef-number reduces with increasing lens translation t_(p) .according toEquation 6.

FIGS. 16 to 20 are photographs taken by a prototype of this invention.These Figures illustrate PSFs observed in mirror ball reflections. Thespheres are placed at increasing distances from the lens (from left tothe right), and are illuminated by a single bright point light source.The PSF due to the translation of a lens and sensor is one-dimensional(1D) because, in this prototype, the translation is restricted to 1D.

FIGS. 16 and 17 are photos that were taken with a static lens. FIG. 16is an all-in-focus photograph taken with a static lens with an f/22aperture (approximating a pinhole). FIG. 17 is a photograph taken with astatic lens with an f/2.8 aperture, focused in the center.

For the photos in FIGS. 18, 19 and 20, the lens and sensor weretranslated relative to the scene during the exposure. The photo in FIG.18 was taken with a lens with an f/22 aperture; whereas the photo inFIG. 18 was taken with a lens with an f/2.8 aperture; focused in thecenter. For the photo in FIG. 20, a vertical slit aperture was used.

According to principles of this invention, physical and virtual blursmay be made orthogonal in order to produce strongly depth-dependent PSF.For example, a vertical slit was used when taking the photo in FIG. 20,in order to create orthogonal physical and virtual blurs.

The photo in FIG. 20 shows strong astigmatism. The PSF in that photochanges from horizontal for points close to the camera (due to thevirtual aperture) to vertical for points further away (due to thephysical lens). In exemplary implementations of this invention, the samelens that is used to take a regular photo may also be translated to takean astigmatic photo, by simply changing the v_(s)/v_(p) ratio. This isan advantage over conventional aspheric lens, which cannot be used totake regular photos.

The photos in FIGS. 21 and 22 were taken by a prototype of thisinvention. They show the same toy figures as FIGS. 12 to 15. FIG. 21 wastaken with a static lens with an f/2.8 aperture. For the photo in FIG.22, a lens and sensor were translated (relative to the scene) during theexposure. Synchronized translation of the lens and sensor simulates theeffect of a larger virtual aperture. The depth of field is shallower,and the bokeh is visually pleasing both in front of and behind the planeof focus. The coordinated translation (of sensor and lens) effectivelyapplies a low-pass filter that removes high-frequency artifacts due tospherical aberration.

The photos in FIGS. 23 and 24 were taken by a prototype of thisinvention. They show the same toy figures as FIGS. 12 to 15.

For the photos in FIGS. 23 and 24, an approximately depth-invariant blursize was achieved by matching the physical blur kernel due to the lensaperture and the virtual blur kernel due to translating lens and sensor.A horizontal slit was placed on the lens to make the PSF purely onedimensional. The lens was physically focused on the closest figure fromthe camera, and the virtual focal plane was at the farthest figure. Asshown in FIG. 23, the resulting blur size was approximatelydepth-invariant. This allowed the application of non-blind imagedeconvolution. The photo in FIG. 24 is an example of the results of suchdeconvolution. For that photo, Richardson-Lucy deconvolution wasemployed to recover an approximately all-in-focus image.

In exemplary implementations of this invention, the defocus effects maybe programmable.

This invention may be implemented in ways other than the examplesdescribed above.

For example, rather than have a lens and sensor move at substantiallyconstant velocities, the velocity profiles of the lens and sensor may bevaried over time. These variations in velocity may be employed to shapePSF and to control defocus (bokeh) characteristics. Also, non-planarfocal surfaces may be obtained using non-linear motion of a sensor andlens.

Also, for example, in a prototype discussed above, the movement of alens and sensor are one dimensional (1D). These movements may instead bytwo-dimensional, such as in a circular, elliptical, hypocycloidal orspiral trajectory. In some implementations, limited sampling may be donefor certain 2D motions.

Also, for example, this invention may be implemented in such a way thatactuators in a cell phone camera (or small point-and-shoot camera) movea lens and a sensor in the camera, during an exposure.

Also, for example, this invention may be implemented with actuators of atype used for image stabilization in existing cameras.

Also, for example, this invention may be implemented by simultaneously(a) moving the camera body relative to the scene being imaged and (b)moving either the sensor or the lens (but not both the sensor and thelens) relative to the camera body and also relative to the scene beingimaged.

CONCLUSION

While a preferred embodiment is disclosed, many other implementationswill occur to one of ordinary skill in the art and are all within thescope of the invention. Each of the various embodiments described abovemay be combined with other described embodiments in order to providemultiple features. Furthermore, while the foregoing describes a numberof separate embodiments of the apparatus and method of the presentinvention, what has been described herein is merely illustrative of theapplication of the principles of the present invention. Otherarrangements, methods, modifications, and substitutions by one ofordinary skill in the art are therefore also considered to be within thescope of the present invention, which is not to be limited except by theclaims that follow.

1. A camera that includes one or more actuators for causing a lens and asensor of said camera, but not said camera as a whole, to move relativeto the scene being imaged, at the same time that the camera captures animage.
 2. The camera of claim 1, wherein the plane of said sensor, theplane of said lens, the direction of motion of said sensor, and thedirection of motion of said lens are all substantially parallel to eachother.
 3. The camera of claim 1, wherein the plane of said sensor issubstantially parallel to the plane of said lens but is notsubstantially parallel to the direction of motion of said lens.
 4. Thecamera of claim 1, wherein said one or more actuators are adapted formoving said lens and said sensor in such a way as to simulate a largeraperture size than the actual aperture size of said lens.
 5. The cameraof claim 1, wherein said one or more actuators are adapted for movingsaid lens and said sensor in such a way as to achieve a substantiallydepth-independent defocus blur size over a range of depths.
 6. Thecamera of claim 5, wherein said substantially depth-independent defocusblur size is achieved over a range of depths while said lens and saidsensor travel at substantially constant velocities, which range extendsbetween the depth at which said lens and said sensor would capture anin-focus image while stationary and the depth at which, if a pinholewere substituted for said lens, said pinhole and said sensor wouldcapture an in-focus image while traveling at said velocities.
 7. Thecamera of claim 1, wherein at least one of said actuators is a steppermotor.
 8. The camera of claim 1, wherein at least one of said actuatorsis piezoelectric.
 9. The camera of claim 1, wherein at least one of saidactuators is ultrasonic.
 10. The camera of claim 1, wherein at least oneof said actuators is further adapted for moving at least one lens orsensor of said camera under certain circumstances, in such a way as tocompensate for motion of said camera.
 11. The camera of claim 1, whereinsaid one or more actuators are adapted for moving said lens and saidsensor, each at a constant velocity for a substantial portion of thetotal time of said movement.
 12. The camera of claim 1, wherein said oneor more actuators are adapted for moving said lens and said sensor, eachat a velocity that varies substantially during a substantial portion ofsaid movement, which portion does not include the initial accelerationor final deceleration that occur during said movement.
 13. The camera ofclaim 1, wherein the motion of said lens or said sensor is circular,elliptical, hypocycloidal or spiral.
 14. The camera of claim 1, whereinsaid image is captured during a single exposure.
 15. The camera of claim1, wherein said movement of said lens and said sensor is programmable.16. A method in which at least one actuator of a camera moves a lens anda sensor of a camera, but not the housing of a camera, relative to thescene being imaged, at the same time that the camera captures an image.17. The method of claim 16, wherein said movement of said lens and saidsensor are programmable.
 18. The method of claim 16, wherein said lensand said sensor are moved in such a way as to simulate a larger aperturesize than the actual aperture size of said lens.
 19. The method of claim16, wherein at least one said actuator moves said lens and said sensorin such a way as to achieve a substantially depth-independent defocusblur size over a range of depths, which range extends between the depthat which said lens and said sensor would capture an in-focus image whilestationary and the depth at which, if a pinhole were substituted forsaid lens, said pinhole and said sensor would capture an in-focus imagewhile traveling at said velocities.
 20. The method of claim 16, whereinat least one actuator of said camera moves at least one lens or sensorof said camera in such a way as to compensate for motion of said camera.